The rise in electronic and digital device technology has rapidly changed the way society communicates, interacts, and consumes goods and services. Modern computing devices allow organizations and users to have access to a variety of useful applications in many locations. Using such applications results in the generation of a large amount of data. Storing and retrieving the produced data is a significant challenge associated with providing useful applications and devices.
The data generated by online services and other applications can be stored at data storage facilities. As the amount of data grows, having a plurality of users sending and requesting data can result in complications that reduce efficiency and speed. Quick and reliable access in storage systems is important for good performance.
Distributed encoded storage systems typically divide each data object to be stored into a plurality of encoded fragments, which are spread across multiple backend storage elements. A distributed encoded storage system maintains metadata which identifies each stored data object, specifies where in the system each data object is stored, including where the encoded data fragments have been distributed and hence from where they can subsequently be retrieved, what type of encoding has been used, etc. For each encoded fragment of the data object metadata such as an identifier, location information and encoding information are maintained. Thus, storage of a single data object generates a large amount of associated metadata.
So that the encoded fragments of a data object can be retrieved during a read operation, conventionally this metadata is maintained and managed by the distributed encoded storage system as a low level data structure. For example, this can be in the form of a hierarchical data structure such as a suitable directory tree of a file system, with corresponding metadata relating a particular data object identifier to the location in the directory tree of encoded fragments.
As noted above, under distributed encoded storage systems, such as for example erasure encoded distributed storage systems, each data object generates a large number of encoded data fragments which need to be stored in the backend. As the number of data objects being stored grows, for example to over one hundred million data objects, the size of the directory tree (or other data structure) of the file system and the corresponding metadata tends to grow beyond the maximum size which can be stored in the main memory of the encoding servers and/or the backend storage entities, and thus needs to be cached to storage media. This leads to an increased level of random input/output operations per second (“IOPS”), in addition to the already high level of TOPS being generated by the high number of encoded fragments for each data object. Conventionally, this can give rise to, e.g., three or four random input/output operations per encoded fragment during a backend read or write operation. This leads to a reduced level of responsiveness of the distributed encoded storage system, and is not generally compatible with storage media that are optimized for sequential storage, such as for example shingled magnetic recording (“SMR”) disks.
It would be desirable to address at least these issues.